<html>
 <head>
  <meta charset="UTF-8">
 </head>
 <body>
  <h2 data-lake-id="bH4t7" id="bH4t7"><span data-lake-id="ub4c25e43" id="ub4c25e43">问题现象</span></h2>
  <p data-lake-id="ucf4fecd0" id="ucf4fecd0"><span data-lake-id="ua2c409bd" id="ua2c409bd">负责的业务中有一个应用因为特殊原因，需要修改消息配置（将Spring Cloud Stream 改为 RocketMQ native），修改前和修改后的配置项如下：</span></p>
  <pre lang="java"><code>
spring.cloud.stream.bindings.consumerA.group=CID_CONSUMER_A
spring.cloud.stream.bindings.consumerA.contentType=text/plain
spring.cloud.stream.bindings.consumerA.destination=CONSUMER_A_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerA.consumer.tags=CONSUMER_A_TOPIC_TAG

spring.cloud.stream.bindings.consumerB.group=CID_CONSUMER_A
spring.cloud.stream.bindings.consumerB.contentType=text/plain
spring.cloud.stream.bindings.consumerB.destination=CONSUMER_B_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerB.consumer.tags=CONSUMER_B_TOPIC_TAG

spring.cloud.stream.bindings.consumerC.group=CID_CONSUMER_A
spring.cloud.stream.bindings.consumerC.contentType=text/plain
spring.cloud.stream.bindings.consumerC.destination=CONSUMER_C_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerC.consumer.tags=CONSUMER_C_TOPIC_TAG
</code></pre>
  <pre lang="java"><code>
spring.rocketmq.consumers[0].consumer-group=CID_CONSUMER_A
spring.rocketmq.consumers[0].topic=CONSUMER_A_TOPIC
spring.rocketmq.consumers[0].sub-expression=CONSUMER_A_TOPIC_TAG
spring.rocketmq.consumers[0].message-listener-ref=consumerAListener

spring.cloud.stream.bindings.consumerB.group=CID_CONSUMER_A
spring.cloud.stream.bindings.consumerB.contentType=text/plain
spring.cloud.stream.bindings.consumerB.destination=CONSUMER_B_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerB.consumer.tags=CONSUMER_B_TOPIC_TAG

spring.cloud.stream.bindings.consumerC.group=CID_CONSUMER_A
spring.cloud.stream.bindings.consumerC.contentType=text/plain
spring.cloud.stream.bindings.consumerC.destination=CONSUMER_C_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerC.consumer.tags=CONSUMER_C_TOPIC_TAG
</code></pre>
  <p data-lake-id="u165409e7" id="u165409e7"><span data-lake-id="u35e0d150" id="u35e0d150">但是当机器发布一半后开始灰度观察的时候，出现了消息堆积问题：</span></p>
  <p data-lake-id="u66080ab4" id="u66080ab4"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682427421166-51c052e1-7992-4ed3-b71e-21d9bdd0709a.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_24%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <h2 data-lake-id="tAhbc" id="tAhbc"><span data-lake-id="ub414ce51" id="ub414ce51">问题原因</span></h2>
  <h3 data-lake-id="hiUTa" id="hiUTa"><span data-lake-id="ubfb1cc5b" id="ubfb1cc5b">消息订阅关系不一致</span></h3>
  <p data-lake-id="ucbb1614a" id="ucbb1614a"><span data-lake-id="u66e600a0" id="u66e600a0">经过历史经验和踩坑，感觉有可能是订阅组机器订阅关系不一致导致的消息堆积问题（因为订阅组的机器有的订阅关系是A，有的是B，MQ不能确定是否要消费，就能只能先堆积到broker中），查看MQ控制台后发现，确实是消息订阅关系不一致，导致消息堆积<br></span><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682427825233-7e043a14-1dac-46b2-9c97-b34d090890e7.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_32%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u56d5cb92" id="u56d5cb92"><span data-lake-id="udf82f389" id="udf82f389">已经发布的那台订阅如下：</span></p>
  <p data-lake-id="u6b0cb92d" id="u6b0cb92d"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682428293109-62907d55-e0c5-469c-abbc-08f3e6cf3b38.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_75%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u4c3f2480" id="u4c3f2480"><span data-lake-id="ua39d4feb" id="ua39d4feb">未发布的订阅关系如下（明显多于已经发布的机器的订阅关系）</span></p>
  <p data-lake-id="ue63e85a6" id="ue63e85a6"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682428655336-a8380f7b-a0ee-4319-8de1-32d9c41144b2.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_67%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <h3 data-lake-id="xJaTE" id="xJaTE"><span data-lake-id="u684b7814" id="u684b7814">Spring Cloud Stream 和 RocketMQ Native</span></h3>
  <p data-lake-id="u2bc5cc12" id="u2bc5cc12"><span data-lake-id="u5e448cc7" id="u5e448cc7">所以就引申出了一个问题，为什么将Spring Cloud Stream修改为原生的MetaQ之后，同一个ConsumerId对应的订阅关系就会改变呢？</span></p>
  <p data-lake-id="u3b9fb1e1" id="u3b9fb1e1"><span data-lake-id="u884aa4c0" id="u884aa4c0">更简单来说，就是为什么当RocketMQ和Spring Cloud Stream 使用相同的ConsumerId之后，RocketMQ的订阅关系会把Spring Cloud Stream的订阅关系给冲掉呢？</span></p>
  <blockquote data-lake-id="u51d48c71" id="u51d48c71">
   <p data-lake-id="ucaf0148f" id="ucaf0148f"><span data-lake-id="u496310f0" id="u496310f0">注意，一个consumerId是可以订阅多个topic的</span></p>
  </blockquote>
  <p data-lake-id="u4dc11dc5" id="u4dc11dc5"><span data-lake-id="u31b704ae" id="u31b704ae">这个时候就只能翻Spring Cloud Stream 和 RocketMQ 的启动源码来解答疑惑。</span></p>
  <h4 data-lake-id="Y0tAR" id="Y0tAR"><span data-lake-id="u6c04a45e" id="u6c04a45e">RocketMQ</span></h4>
  <p data-lake-id="u1653b36e" id="u1653b36e"><span data-lake-id="u4ac8a1a9" id="u4ac8a1a9">RocketMQ client的类图如下：</span></p>
  <p data-lake-id="u97881eb3" id="u97881eb3"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425441600-dea2f79c-1e75-497a-b208-f6bf09973596.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_36%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <ul list="uf0dab954">
   <li fid="ub89bd4fc" data-lake-id="u1e3bd72e" id="u1e3bd72e"><span data-lake-id="uc2ecd362" id="uc2ecd362">MQConsumerInner：记录当前consumerGroup和服务端的交互方式，以及topic和tag的映射关系。默认的实现是DefaultMQPushConsumerImpl，和consumerGroup的对应关系是1 : 1</span></li>
   <li fid="ub89bd4fc" data-lake-id="u3ce29fc9" id="u3ce29fc9"><span data-lake-id="u656bc608" id="u656bc608">MQClientInstance：统一管理网络链接等可以复用的对象，通过Map维护了ConsumerGroupId和MQConsumerInner的映射关系。简单来说，</span><span data-lake-id="u1afd9ec4" id="u1afd9ec4" style="color: #DF2A3F">就是一个ConsumerGroup，只能对应一个MQConsumerInner，</span><span data-lake-id="u05f36880" id="u05f36880">如下代码所示：</span></li>
  </ul>
  <p data-lake-id="u0e34a010" id="u0e34a010"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425442602-e5ee0fff-4b44-4ae0-aeae-8f563baf3942.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_40%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <h4 data-lake-id="Cl9OB" id="Cl9OB"><span data-lake-id="u5b326b10" id="u5b326b10">Spring Cloud Stream</span></h4>
  <p data-lake-id="ubc8ca3e9" id="ubc8ca3e9"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425443533-40b769ce-8a26-481d-a42e-26a2949fd9fe.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_11%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="udc35df70" id="udc35df70"><span data-lake-id="u58a9e31a" id="u58a9e31a">Spring Cloud Stream是连接Spring和中间件的一个胶水层，在Spring Cloud Stream启动的时候，也会注册一个ConsumerGourp，如下代码所示：</span></p>
  <p data-lake-id="ue45ed688" id="ue45ed688"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425444585-ca1490b9-08c7-4654-98a3-726304f6e095.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_71%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <h3 data-lake-id="xUlBK" id="xUlBK"><span data-lake-id="u3b166279" id="u3b166279">问题根因</span></h3>
  <p data-lake-id="u3eff8d54" id="u3eff8d54"><span data-lake-id="u72932ba3" id="u72932ba3">分析到这里，原因就已经很明显了。Spring Cloud Stream会在启动的时候自己new一个MetaPushConsumer（事实上就是一个新的MQConsumerInner），所以对于一个ConsumerGroup来说，就存在了两个MQConsumerInner，这显然是不符合RocketMQ要求的1:1的映射关系的，所以RocketMQ默认会用新的映射代替老的映射关系。显然，Spring Cloud Stream的被RocketMQ原生的给替代掉了。</span></p>
  <p data-lake-id="ubbe68c84" id="ubbe68c84"><span data-lake-id="u6c6befb0" id="u6c6befb0">这也就是为什么已经发布的机器中，对于ConsumerA来说，只剩下RocketMQ原生的那组订阅关系了</span></p>
  <h2 data-lake-id="siVLs" id="siVLs"><span data-lake-id="ue6f61baf" id="ue6f61baf">解决思路</span></h2>
  <p data-lake-id="ue11f37e0" id="ue11f37e0"><span data-lake-id="u9cf34066" id="u9cf34066">修改consumerId</span></p>
  <pre lang="java"><code>
spring.rocketmq.consumers[0].consumer-group=CID_CONSUMER_A
spring.rocketmq.consumers[0].topic=CONSUMER_A_TOPIC
spring.rocketmq.consumers[0].sub-expression=CONSUMER_A_TOPIC_TAG
spring.rocketmq.consumers[0].message-listener-ref=consumerAListener

spring.cloud.stream.bindings.consumerB.group=CID_CONSUMER_B
spring.cloud.stream.bindings.consumerB.contentType=text/plain
spring.cloud.stream.bindings.consumerB.destination=CONSUMER_B_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerB.consumer.tags=CONSUMER_B_TOPIC_TAG

spring.cloud.stream.bindings.consumerC.group=CID_CONSUMER_B
spring.cloud.stream.bindings.consumerC.contentType=text/plain
spring.cloud.stream.bindings.consumerC.destination=CONSUMER_C_TOPIC
spring.cloud.stream.rocketmq.bindings.consumerC.consumer.tags=CONSUMER_C_TOPIC_TAG
</code></pre>
  <h2 data-lake-id="c9ziy" id="c9ziy"><span data-lake-id="u6adc92b3" id="u6adc92b3">思考和总结</span></h2>
  <ol list="u61ac04f3">
   <li fid="u7b8b305f" data-lake-id="u698f0575" id="u698f0575"><span data-lake-id="uf8c66268" id="uf8c66268">问题原因并不复杂，但是很多人可能分析到第一层（订阅关系不一致导致消费堆积）就不会再往下分析了，但是我们还需要有更深入的探索精神的</span></li>
   <li fid="u7b8b305f" data-lake-id="u863c3730" id="u863c3730"><span data-lake-id="u85e10ebb" id="u85e10ebb">生产环境中尽量不要搞两套配置项，会额外增加理解成本。。。。</span></li>
  </ol>
  <h2 data-lake-id="TjCJ3" id="TjCJ3"><span data-lake-id="ub0110136" id="ub0110136">小技巧</span></h2>
  <h3 data-lake-id="xeWag" id="xeWag"><span data-lake-id="udf994cdb" id="udf994cdb">中间件代码如何确定版本</span></h3>
  <p data-lake-id="ue0fe2b2a" id="ue0fe2b2a"><span data-lake-id="u7a1c912c" id="u7a1c912c">arthas中的sc 命令 </span></p>
  <p data-lake-id="u2822f209" id="u2822f209"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425445317-52934b47-4f0c-4ba9-9787-adf6713e075b.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_73%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <h3 data-lake-id="cL6Ot" id="cL6Ot"><span data-lake-id="u79983c1c" id="u79983c1c">Idea如何debug具体版本的中间件</span></h3>
  <p data-lake-id="u08e2fed8" id="u08e2fed8"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425445693-f30c71da-02ab-4d5b-bf80-6ecb10d1a66d.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_51%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
  <p data-lake-id="u99625f98" id="u99625f98"><br></p>
  <p data-lake-id="uf24537f0" id="uf24537f0"><img src="https://cdn.nlark.com/yuque/0/2023/png/719664/1682425446186-55f31009-2326-4304-903e-36954e97eaec.png?x-oss-process=image%2Fwatermark%2Ctype_d3F5LW1pY3JvaGVp%2Csize_81%2Ctext_SmF2YSA4IEd1IFA%3D%2Ccolor_FFFFFF%2Cshadow_50%2Ct_80%2Cg_se%2Cx_10%2Cy_10"></p>
 </body>
</html>